The Merits of Externally Invalid Survey Experiments

 

Gustavo Diaz
McMaster University
gustavodiaz.org

 

Slides: gustavodiaz.org/talk

Limitations

“Future research should confirm if our findings generalize…”

  • …with a representative sample
  • …in other countries
  • …beyond the survey setting
  • …when using behavioral outcomes

Usual workflow

  1. Research idea

  2. Realize resource/ethical/practical limitations

  3. Conduct experiment with limitations

  4. Wave hands about external validity

Goal

  • Should we ever implement an externally invalid survey experiment on purpose?

  • Identify what makes external invalidity desirable

  • Challenge: Different kinds of external (in)validity

External validity concerns

Type Concern
Samples Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

External validity concerns

Type Concern
Samples Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

External validity concerns

Type Concern
Samples Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

External validity concerns

Type Concern
Samples Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

External validity concerns

Type Concern
Samples Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

Examples

Samples

Samples

Treatments

Treatments

Outcomes

Outcomes

Contexts

Contexts

Saudi Arabia and Kuwait were selected for their theoretical case value;

Contexts

Saudi Arabia and Kuwait were selected for their theoretical case value; both are high in gender inegalitarianism, and they offer tough tests.

Contexts

Saudi Arabia and Kuwait were selected for their theoretical case value; both are high in gender inegalitarianism, and they offer tough tests. In addition, while these neighboring countries have much in common, both resource-rich and highly conservative, they also differ in important ways.

Contexts

Saudi Arabia and Kuwait were selected for their theoretical case value; both are high in gender inegalitarianism, and they offer tough tests. In addition, while these neighboring countries have much in common, both resource-rich and highly conservative, they also differ in important ways. Thus, if similar results are found, the case for generalizability across different interaction types and varying national circumstances will be strengthened.

Conclusion

Invalid Benefit
Samples Delineate existing generalizations
Treatments Statistical properties
Outcomes Hypothetical/rare scenarios
Contexts Delineate existing generalizations

  • Endline: Consider merits while planning experiments

  • What would persuade you to embrace external invalidity?